Generating composite images for display on a mobile device based on ground truth image rendering

ABSTRACT

Systems and methods are disclosed herein for monitoring a location of a vehicle approaching a pickup location associated with a transportation service and generating composite images for display on a driver client device in response to determining that the vehicle is within a threshold distance of a pickup location. The systems and methods use sensor data from the driver client device and a device localization process to monitor the location of the driver client device located in the vehicle. In response to determining that the driver client device (and thus, the vehicle) is within a threshold distance of the pickup location the service generates one or more composite images by overlaying a stored image of the pickup location with one or more persistent augmented reality elements associated with the pickup location and/or the transportation service.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 62/812,098, filed Feb. 28, 2019, U.S. Provisional Application No. 62/795,988, filed Jan. 23, 2019, U.S. Provisional Application No. 62/812,101, filed Feb. 28, 2019, U.S. Provisional Application No. 62/801,010, filed Feb. 4, 2019, U.S. Provisional Application No. 62/801,012, filed Feb. 4, 2019, U.S. Provisional Application No. 62/802,145, filed Feb. 6, 2019, and U.S. Provisional Application No. 62/812,107, filed Feb. 28, 2019, which are incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to location determination with limited or no reliance on global positioning system (GPS) signals, and in particular to determining a location estimate for a client device and generating composite images for display on the client device based on the determined location.

BACKGROUND

Many systems may use global positioning system (GPS) coordinates to estimate the position of persons carrying client devices. For example, a person (referred to herein as a “rider”) carrying a client device may wish to arrange for transportation from his or her present location to another location, and may execute an application (e.g., a transportation service and/or ridesharing application) on his or her client device to obtain transportation from a transportation provider (referred to herein as a “driver”). Existing systems in this scenario match the rider with a driver, and instruct the driver to travel to a location of the rider for pickup. However, the GPS coordinates used to estimate the position of the rider may be unknown or inaccurate due to interference or reception problems, such as those caused by tall buildings that distort satellite signals, for example. This erroneous GPS data (e.g., inaccurate or missing data) may cause inconveniences for the driver when attempting to locate the rider for pickup, and/or designating a pickup location altogether. Similarly, this scenario may cause inconveniences for the rider while waiting for the driver to travel to the pickup location given the potentially inaccurate data and cause the rider to consider alternate applications that offer rideshare services. Furthermore, it may be difficult for the driver to identify the rider's actual location, particularly in congested areas with heavy foot traffic, which may lead to inefficiencies in the pickup process, cause frustration to the parties, and delay the transportation service.

Existing systems seek to solve the technical problem of locating a rider and identifying the rider to the driver by prompting riders to manually confirm their locations within the application. However, these systems can be cumbersome for riders that are unfamiliar with their surroundings or unsure of their exact locations. Additionally, many of these systems do not aide the driver in identifying the rider's true location as the driver approaches the pickup location if the GPS traces from the rider's client device, or the rider's confirmed location, are inaccurate. The technical problem of identifying an accurate location of a rider to a driver approaching a pickup location is not addressed by existing systems.

SUMMARY

Systems and methods are disclosed herein for determining a location of a vehicle and generating for display on a driver client device composite images including augmented reality (“AR”) elements associated with a transportation service. To this end, a service (e.g., that connects a rider with a driver in the context of a transportation application) receives a request from a user of the service for transportation to a destination location. In response to receiving the request, the service matches the requesting user with one of a plurality of available drivers based in part on the respective locations of the rider and driver. If the matched driver accepts an invitation from the service to fulfill the request, the service monitors the location of the driver's vehicle as the driver approaches a pickup location for the transportation service.

In one embodiment, the service initializes the determination of the vehicle location based on global positioning system (GPS) data of the driver client device, and continues to use GPS traces as a proxy for the driver's location until the driver is in the vicinity of the pickup location. In response to determining that the vehicle is within a first threshold distance of the pickup location, to counter the possibility that the driver's GPS traces are inaccurate (e.g., due to being within an urban canyon), the service uses a device localization process to further monitor the vehicle location by comparing renderings of images captured by the driver client device to candidate images corresponding to the pickup location. In response to the captured image matching at least one of the candidate images, the service determines that the vehicle is within a second threshold distance of the pickup location and generates for display on the driver client device a composite image including a base image of the pickup location and a persistent AR overlay including one or more AR elements associated with the pickup location and/or the service request.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a location estimation system, according to one embodiment.

FIG. 2 is an illustration of GPS traces in a region where GPS signals are inaccurate according to one embodiment.

FIG. 3 is an illustration of a manner in which a vehicle location is initialized and then updated, according to one embodiment.

FIG. 4 is an illustration of a composite image identifying rider and pickup locations on a driver client device, according to one embodiment.

FIG. 5 is an illustrative flowchart of a process for generating composite images for display on a client device based on ground truth image rendering, according to one embodiment.

FIG. 6 is a block diagram that illustrates a computer system, according to one embodiment.

The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

DETAILED DESCRIPTION System Environment

FIG. 1 is a block diagram illustration a location estimation system, according to one embodiment. System 100 includes vehicle 101, which contains or is carrying client device 110. The functionality of client device 110 is described in further detail with respect to FIG. 6 below. In some embodiments, client device 110 is integrated into vehicle 101 as a component of vehicle 101. Client device 110 executes an application, such as a transportation service and/or ridesharing application where a rider may request a ride from the rider's current location to a desired destination, and where the rider may be connected to a driver who also uses the application, where the driver will provide the ride. While the driver travels toward a rider, and while the driver is transporting a rider to a destination, a map may be viewed by the driver or the rider via the application where an indicator of the rider's position is displayed. In an exemplary embodiment, client device 110 is mounted on a dashboard or windshield of vehicle 101 and has a forward-facing camera that faces the road. While this exemplary embodiment is referred to throughout, in some embodiments, the application instead commands images to be captured from a stand-alone camera (e.g., embedded in a device that is affixed to a dashboard or windshield of vehicle 101).

In an embodiment, client device 110 automatically captures one or more images based on commands received from the application. For example, client device 110 captures a frame upon a certain condition being satisfied (e.g., a certain distance has been traveled, or a certain amount of time has passed, from a reference point, or a certain threshold distance of a pickup location has been reached). Times at which images are automatically captured by client device 110 will be described in further detail below with reference to FIGS. 3-5. Automatic capturing of one or more images may be an opt-in feature, where the application by default does not automatically capture images using a camera of client device 110, and where the application has a setting that, if selected by a user of client device 110 (e.g., a driver of vehicle 101), enables the application to automatically capture the images. While accurate pinpointing of a driver's location using the systems and methods described herein may rely on opting in, the location of the driver may be determined based on GPS traces of client device 110 (even if inaccurate) should a driver of vehicle 101 not opt in.

In some embodiments, client device 110 transmits the image(s) to location determination service 130 over network 120, where location determination service 130 receives the image(s) and compares them to known images, stored at image rendering database 132, to determine the location of client device 110. The location determination service 130 compares the determined location to a pickup location associated with a service request, and responsive to detecting that client device 110 is within a second threshold distance of the pickup location, instructs image generation service 134 to generate a composite image of the pickup location for display on client device 110.

In some embodiments, the functionality of location determination service 130, image rendering database 132, and/or image generation service 134 is located within client device 110, and thus need not be accessed by network 120, as depicted. Functionality of location determination service 130 and image generation service 134 may be integrated as one or more modules of the application. Image rendering database 132 may be accessed by location determination service 130 directly, or over network 120. Location determination service 130 and image generation service 134 may be one or more modules of an application, such as a ridesharing application, or may be components of a transportation service generally, such as a ridesharing service. In some embodiments where location determination service 130 and image generation service 134 are modules of the application, some or all of the contents of image rendering database 132 are transmitted to the client device for performing localization at the client device. The functionality of location determination service 130 and image generation service 134 will be described in further detail below with respect to FIGS. 2-6.

Identifying Regions Prone to Erroneous GPS Readings

FIG. 2 is an illustration of GPS traces in a region where GPS signals are inaccurate according to one embodiment. Region 200 includes GPS traces 202 of a client device (e.g., client device 110) as derived from a GPS sensor of client device 110. As an illustrative example, the GPS traces 202 were derived from client device 110 while vehicle 101 was on a road. Because of the existence of tall buildings within region 200, the GPS signals used to derive the GPS traces are distorted and provide inaccurate GPS traces. This is evidenced by the GPS traces being at locations that are not on a road. GPS traces may similarly be unreliable for a client device associated with a user of a ridesharing application who is located at, or traveling to, a pickup location.

Region 200 is exemplary of a location known to location determination service 130 to have or cause erroneous GPS data. The identification of various regions, like region 200, which are associated with erroneous GPS data may be performed automatically by location determination server 130, or may be made based on manual feedback (e.g., performed in advance of executing process 500). For example, location determination service 130 may detect that users of a ridesharing application in a given location set a pickup pin at a location different from their GPS traces at a frequency that exceeds an implementer-defined threshold, and may determine therefrom that GPS data derived from client devices within that region are likely erroneous. As another example, location determination service 130 may detect that GPS traces of users (e.g., drivers) of a ridesharing application are, at a frequency above a threshold, in areas inaccessible to drivers, such as within buildings or parks that do not have road access, and may determine therefrom that GPS data derived from client devices within that region are likely erroneous. As yet another example, location determination service 130 may receive feedback from users that their client devices are determining erroneous locations based on GPS sensors of those client devices, and may determine therefrom that GPS data derived from client devices within that region are likely erroneous.

Exemplary Initialization using GPS and Transition to Image-Based Localization

FIG. 3 is an example of a vicinity encompassing a pickup location, according to one embodiment. In the embodiment shown in FIG. 3, a rider has requested (e.g., using the above-described application) a transportation service to a destination location and a pickup location 300 has been designated as the starting point for the transportation service. In one embodiment, location determination service 130 selects a pickup location for the requesting user and provides the selected pickup location for display on a rider client device. Alternatively, location determination service 130 selects one or more candidate pickup locations within a threshold distance of a current location of a rider client device and provides the candidate pickup options for display on the rider client device. In response to receiving a selection of candidate pickup location from the rider client device, location determination service 130 sets the selected location as the pickup location. In still other embodiments, location determination service 130 may detect that the requesting user set a pickup pin via the application and may assign the associated location of the pickup pin as the pickup location.

In response to receiving the service request, location determination service 130 matches the requesting user with one of a plurality of available drivers. In one embodiment, selection of an available driver is based in part on current locations of driver and rider client devices as determined automatically using global positioning system (GPS) data received from the driver and rider client devices. Location determination service 130 further provides an invitation message to the matched driver inviting the driver to fulfill the service request, and if the driver accepts via input to driver client device 110, location determination service 130 assigns the selected driver to the transportation service.

In response to assigning the selected driver to the transportation service, location determination service 130 initiates monitoring of location data of the driver's vehicle (such as vehicle 101) as vehicle 101 travels from its current location to the pickup location 300. In one embodiment, location determination service 130 periodically retrieves GPS traces acquired using a GPS sensor of client device 110 and compares the retrieved GPS traces to a GPS identifier of the pickup location 300 to determine a position of vehicle 101 relative to the pickup location 300.

Location determination service 130 continues to monitor the location of vehicle 101 using received GPS traces to detect a scenario where vehicle 101 reaches a position (e.g., a position “A” in FIG. 3) within a first threshold distance 310 of the pickup location 300. Detection that vehicle 101 has reached the first threshold distance 310 causes location determination service 130 to trigger a device localization process to continue monitoring the location of vehicle 101 as vehicle 101 approaches the pickup location.

In response to determining that vehicle 101 has reached the first threshold distance 310 (e.g., is at position A), the application instructs client device 110 to capture an image that will be used by location determination service 130 to determine the location of vehicle 101 without further use of a GPS sensor of client device 110. The application causes client device 110 to transmit to location determination service 130 a rendering of the captured image. Location determination service 130 identifies a subset of entries of image rendering database 132 that correspond to a vicinity of the first threshold distance 310 and compares the rendering to renderings of each entry in the subset. For example, keypoints (i.e., spatial locations in the rendering that define areas of interest) of the received rendering may be extracted and compared to keypoints of candidate renderings to determine whether a threshold amount of keypoints match. In some embodiments, in order to perform this determination, location determination service 130 determines that the received rendering does not completely match any entry of the entries. For example, when comparing two-dimensional renderings, location determination service 130 may determine that not all keypoints of the received rendering match any candidate rendering. When comparing three-dimensional renderings, location determination service 130 may determine that the keypoints of the image do not match all keypoints of any perspective of any candidate rendering.

Location determination service 130 may first perform the matching coarsely by leveraging GPS to reduce the search space. By using a large radius around a query image GPS position, location determination service 130 could produce candidate images or 3D sections of the scene against which to match the captured image. Location determination service 130 may further filter candidate images or 3D sections by using the heading direction of the query image or 3D scene coordinates to more readily align them to a “base map” that stitches together known renderings into a model of the world (e.g., a map of a 2D or 3D model of known renderings). Additional techniques such as vocabulary trees, a bag-of-words model, or machine learning can be used to quickly retrieve a matching set of images or 3D content.

In one embodiment, location determination service 130 performs alignment of 3D models to determine a matching rendering. 3D-3D alignment may be performed in various ways. For example, traditional approaches use an iterative closest point (ICP) algorithm. The alignment may be seeded using machine-learning models that semantically segment the 3D scene. The segmentation can be used to create a coarse alignment between similar semantic structures, such as, car-to-car, light post-to-light post, and so forth. After generating the coarse alignment, location determination service 130 may revert to traditional ICP to perform the final precision alignment in an accelerated fashion.

In response to determining that the received rendering does not completely match any entry of the entries, location determination service 130 determines that a percentage of characteristics of the received rendering match characteristics of the given entry, and determines whether the percentage exceeds a threshold. In response to determining that the percentage exceeds the threshold, location determination service 130 determines that the received rendering matches the given entry based on the partial match. Likewise, in response to determining that the percentage does not exceed the threshold, location determination service 130 determines that the received rendering does not match the given entry notwithstanding the partial match.

In response to finding a matching rendering, location determination service 130 determines the location of vehicle 101 at a location indicated in the entry that includes the matching rendering (i.e., at a location within the first threshold distance 310). Location determination service 130 continues to monitor the location of vehicle 101 by instructing client device 110 to capture one or more frames upon a certain condition being satisfied (e.g., a certain distance has been traveled or a certain amount of time has passed from a reference point or since a previous location of client device 110 was determined).

Location determination service 130 continues to use image-based localization to monitor the location of vehicle 101 to detect a scenario where vehicle 101 reaches a position (e.g., a position “B” in FIG. 3) within a second threshold distance of the pickup location 300. For example, location determination service 130 compares images captured by client device 110 to stored renderings that each correspond to the pickup location 300 (referred to herein as “candidate images”). If the captured image matches at least one of the candidate images, location determination service 130 determines that vehicle 101 is within the second threshold distance 320 of the pickup location 300.

While the embodiment discussed above utilizes GPS traces to monitor the location of client device 110 until client device 110 reaches the first threshold distance 310, in other embodiments, location determination service 130 performs device localization before the first threshold distance 310 is reached. For example, responsive to detecting that vehicle 101 has entered a region where GPS signals are known to be inaccurate, such as region 200, location determination service 130 triggers device localization regardless of whether vehicle 101 has reached position A.

Responsive to determining that vehicle 101 is within a second threshold distance 320 of the pickup location 300, location determination service 130 instructs image generation service 134 to generate a composite image of the pickup location 300 for display on client device 110. Alternatively, for safety reasons, triggering display of the composite image may be based on determining that vehicle 101 has been stopped for at least a threshold period of time. For example, location determination service 130 may monitor sensor data of client device 110 (e.g., data obtained from an accelerometer of client device 110) and may send the image generation instruction to image generation service 134 in response to determining that vehicle 101 is within the second threshold distance 320 and has been stopped for at least a threshold period of time (e.g., 10 seconds). The threshold period of time may be greater than a time period during which vehicle 101 might be stopped at a stop sign or as a result of sudden braking such that determining that vehicle 101 has been stopped for at least the threshold period of time indicates that vehicle 101 is likely stopped at a traffic signal, at the pickup location, or has otherwise pulled over and can safely view the composite image. In another embodiment, if location determination service 130 detects renewed movement of client device 110 (and thus, vehicle 101) prior to the driver confirming through client device 110 that the pickup has occurred and the requested service has started, location determination service 130 instructs client device 110 to cease display of the composite image. Responsive to detecting a next stop of vehicle 101 for over the threshold period of time, location determination service 130 to resume display of the composite image. In still another embodiment, display of the image is based on user input through client device 110. For example, the driver might request, through the application, an AR option for identifying the rider associated with the service request. Still further, location determination service 130 might provide for display on client device 110 a smaller image of a map of the vicinity of vehicle 101 while vehicle 101 is in motion and may trigger display of the composite image in response to determining that vehicle 101 has been stopped for at least the threshold period of time.

In one embodiment, the composite image includes an image corresponding to the pickup location (the “base image”) overlaid by one or more persistent augmented reality (“AR”) elements associated with the pickup location and/or the service request. The base image may be selected by image generation service 134 based on a line of sight of a camera sensor of client device 110 and/or based a position and angle of client device 110 (e.g., as measured by an inertial measuring unit (IMU) of client device 110). In one embodiment, the base image is the frame captured by client device 110 that matches at least one of the candidate images of the pickup location retrieved from image rendering database 132. Alternatively, image generation service 134 selects a candidate as the base image.

Image generation service 134 generates an overlay of one or more static or dynamic computer-generated elements by identifying, in the base image, at least one visual indicator of the pickup location and/or the rider associated with the service request and places the AR elements on a portion of the base image where the visual indicator is located. For example, in one embodiment, the visual indicator corresponds to the pickup location and the AR elements include an identification of the pickup location, such as a highlighted portion of a road or sidewalk, flag or other graphical marker, text (e.g., text reading “The pickup location is 20 meters ahead,” or “Stop here”), and the like. Similarly, if the rider has opted-in to sharing the rider's location and/or photograph with the driver, image generation service 134 determines whether the base image includes an identification of the rider (e.g., if the base image is an image captured by client device 110) using stored data associated with the rider (e.g., the rider's photograph) and/or location data associated with a rider client device. As will be discussed below with respect to FIGS. 4 and 6, if image generation service 134 identifies the rider and/or rider client device in the base image, image generation service 134 may generate one or more AR elements associated with the rider, such as an arrow or other graphical marker pointing to the rider, text identifying the rider, and the like. Image generation service 314 then superimposes the AR elements on a position of the base image where the visual indicator is located to generate the composite image. The composite image is then provided for display on client device 110. In some embodiments, image generation service 134 generates the composite image in response to location determination service 130 determining that vehicle 101 is within the second threshold distance 320 of the pickup location 300, but does not provide the composite image for display on client device 110 until location determination service 130 determines that vehicle 101 is stopped.

While the embodiment described above generates a composite image responsive to determining that vehicle 101 is within a second threshold distance of the pickup location, in some embodiments, updated composite images may be generated responsive to determining that vehicle 101 has reached one or more subsequent thresholds. Image generation service 134 may continuously generate and provide to client device 110 updated composite images as vehicle 101 moves toward the pickup location such that the images are displayed in a video-like fashion on client device 110.

Exemplary Display of Augmented Reality Elements on Driver Device

FIG. 4 is an illustration of a composite image 402 displayed on client device 110. In one embodiment, client device 110 is mounted (e.g., on the dashboard or windshield) of vehicle 101. Alternatively, client device 110 may be associated with a heads-up display unit that displays AR elements in the driver's line of sight.

In response to determining that vehicle 101 is within the second threshold distance of the pickup location, location determination service 130 instructs image generation service 134 to generate a composite image of the vicinity of the pickup location. The details of the device localization process are described above with respect to FIG. 3, the details of which apply fully hereto. In response to receiving the image generation instruction, image generation service 134 selects a base image of the pickup location for display on client device 110 and generates one or more persistent static and/or dynamic AR elements associated with the pickup location and/or the service request.

In one embodiment, image generation service 134 generates the AR elements using the camera direction and tilt of client device 110. For example, if location determination service 130 determines that the pickup location is within a line of sight of a camera sensor of client device 110, image generation service 134 generates one or more AR elements to be superimposed on the base image at positions of the base image corresponding to visual indicators of the pickup location and/or the requesting rider. In the displayed composite image 402, client device 110 displays an image of a road approaching a pickup location. The base image is overlaid with graphical and textual elements identifying the visual indicators (e.g., the rider and the pickup location). For example, composite image 402 includes an arrow 404 pointing to the pickup location and a text box 406 including the rider's name and rating, and the distance from client device 110 to the rider's position. Responsive to determining that the rider has moved (e.g., based on data received from the rider client device), image generation service 134 generates an updated composite image for display on client device 110 showing an updated rider location. In this way, real-time or near real-time location changes of the rider may be shown with AR overlays on client device 110. In various embodiments, the text box 406 includes different information about the rider based on rider-specified settings. Additionally or alternatively, the AR elements may include graphical and/or textual elements identifying the pickup location, such as a highlighted portion of the road, sidewalk, and/or other element of the base image associated with the pickup location (e.g., a pole, a sign, a tree, and the like).

Exemplary Composite Image Generation for Display on Driver Device

FIG. 5 is an illustrative flowchart of a process for generating a composite image of a pickup location for display on a client device (e.g., client device 110 within vehicle 101), in accordance with some embodiments of the disclosure. A service, (e.g., a service that connects a rider with a driver in the context of a transportation application) receives a request for a ride from a user client device. Upon receiving the request, location determination service 130 identifies a location of the user and assigns a driver to service the service request based in part on a comparison of the user's location and the locations of a plurality of candidate drivers. Process 500 begins by the service receiving 502 a confirmation from a client device (e.g., client device 110 within vehicle 101) of the assigned driver that the driver will navigate to a pickup location associated with the service request.

As the driver travels to the designated pickup location, location determination service 130 monitors 504 the location of client device 110 determine when client device 110 is within first and second threshold distances of the pickup location. The details of the device monitoring process are described above with respect to FIG. 3, the details of which apply fully hereto. For example, in some embodiments, location determination service 130 initially monitors the location of client device 110 using GPS sensor data of client device 110, and, responsive to determining that client device 110 (and thus, vehicle 101) is within a first threshold distance of the pickup location, triggers a device localization process to further monitor the location of vehicle 101.

Location determination service 130 continues to monitor the location of client device 110 as vehicle 101 approaches the pickup location to determine 506 whether client device 110 is within a second threshold distance of the pickup location. Renderings of images captured by client device 110 are compared to a plurality of candidate images corresponding to the pickup location. If the rendering of the captured image does not match at least one of the candidate images, location determination service 130 continues to monitor the location of client device 110 until location determination service 130 determines that an image captured by client device 110 matches a candidate image. In response to determining that the captured rendering matches at least one candidate image, location determination service 130 determines that the location of vehicle 101 is the location associated with the matching rendering and that vehicle 101 is within a second threshold distance of the pickup location.

In response to determining that client device 110 is located within the second threshold distance of the pickup location (and, optionally, in response to determining that vehicle 101 has been stopped for at least a threshold period of time, e.g., based on an accelerometer of client device 110), location determination service 130 instructs image generation service 134 to generate a composite image of the pickup location for display on client device 110. Process 500 continues with image generation service 134 selecting 508 an image of a vicinity of the pickup location (the “base image”). In one embodiment, image generation service 134 selects as the base image the frame captured by client device 110 that matches at least one candidate image. Alternatively, the matching candidate image is selected as the base image. In still other embodiments, image generation service 134 selects a different image of the pickup location stored in image rendering database 132 based in part on a direction that client device 110 is facing and/or a portion of the pickup location that is within a line of sight of a camera sensor of client device 110.

Image generation service 134 generates the composite image by identifying, in the base image, at least one visual indicator of the pickup location and/or the requesting rider, generating at least one persistent AR element associated with the visual indicator, and superimposing the generated element on a portion of the base image where the visual indicator is located. In one embodiment, image generation service 134 determines whether the rider associated with the service request has consented to sharing the rider's location and/or photograph with the driver. If image generation service 134 determines that the rider has consented to such data sharing and if image generation service 134 determines that the rider is visible in the base image (e.g., in instances where the base image is an image captured by client device 110), image generation service 134 generates one or more AR elements identifying the rider for display on client device 110. Additionally or alternatively, image generation service 134 generates one or more graphical and/or textual AR elements identifying the pickup location, as discussed above with respect to FIGS. 3 and 4. Image generation service 134 overlays the one or more AR elements onto the base image to generate the composite image and instructs client device 110 to generate for display the composite image.

Computing Hardware

The entities shown in FIG. 1 are implemented using one or more computers. FIG. 6 is a block diagram that illustrates a computer system 600 for acting as a client 110 or location determination service 130, according to one embodiment. Illustrated are at least one processor 602 coupled to a chipset 604. Also coupled to the chipset 604 are a memory 606, a storage device 608, a keyboard 610, a graphics adapter 612, a pointing device 614, and a network adapter 616. A display 618 is coupled to the graphics adapter 612. In one embodiment, the functionality of the chipset 604 is provided by a memory controller hub 620 and an I/O controller hub 622. In another embodiment, the memory 606 is coupled directly to the processor 602 instead of the chipset 604.

The storage device 608 is any non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device. The memory 606 holds instructions and data used by the processor 602. The pointing device 614 may be a mouse, track ball, or other type of pointing device, and is used in combination with the keyboard 610 to input data into the computer system 600. The graphics adapter 612 displays images and other information on the display 618. The network adapter 616 couples the computer system 600 to the network 120.

As is known in the art, a computer 600 can have different and/or other components than those shown in FIG. 6. In addition, the computer 600 can lack certain illustrated components. For example, the computer acting as the location determination service 130 can be formed of multiple blade servers linked together into one or more distributed systems and lack components such as keyboards and displays. Moreover, the storage device 608 can be local and/or remote from the computer 600 (such as embodied within a storage area network (SAN)).

Additional Considerations

The foregoing description described one embodiment of the invention in which a central server including location determination service 130, image rendering database 132, and image generation service 134 monitors the location of a client device (such as client device 110) and generates at least one composite image for display on the client device. In other embodiments, some or all of the functions described from the perspective of the central server are performed on the client device. For example, location determination service may be a module installed in a transportation application executing on the client device. Additionally, while the embodiment described herein generates composite images comprising a base image overlaid with one or more computer-generated AR elements, in other embodiments, the AR elements are overlaid on a video feed captured by a camera of client device 110. Still further, while the described embodiment describes a transportation service provider as a driver of a vehicle, in other embodiments, the transportation service provider is an autonomous vehicle that transports the rider to the destination location.

The foregoing description has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations while described functionally computationally or logically are understood to be implemented by computer programs or equivalent electrical circuits microcode or the like. Furthermore, it has also proven convenient at times to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules may be embodied in software firmware hardware or any combinations thereof.

Any of the steps operations or processes described herein may be performed or implemented with one or more hardware or software modules alone or in combination with other devices. In one embodiment a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code which can be executed by a computer processor for performing any or all of the steps operations or processes described.

Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer readable storage medium or any type of media suitable for storing electronic instructions which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process where the information is stored on a non-transitory tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative but not limiting of the scope of the invention which is set forth in the following claims. 

1. A computer-implemented method for generating images associated with a transportation service, the method comprising: receiving confirmation from a client device of a driver that the driver will navigate to a pickup location; in response to receiving the confirmation, monitoring a location of the client device to determine whether the client device is within a threshold distance of the pickup location; determining, based on the monitoring, that the client device is located with the threshold distance of the pickup location; in response to determining that the client device is located within the threshold distance of the pickup location, selecting, based in part on a direction that the client device is facing, an image of one or more candidate images that each correspond to the pickup location; and instructing the client device to generate for display the image.
 2. The method of claim 1, further comprising: generating persistent augmented reality content for display on the client device, the persistent augmented reality content associated with the pickup location; and instructing the client device to display the persistent augmented reality content as an overlay on top of the image.
 3. The method of claim 1, wherein determining that the client device is within the threshold distance of the pickup location comprises: responsive to an occurrence of a condition, instructing the client device to capture an image as the client device approaches the pickup location; comparing a rendering of the captured image to the plurality of candidate images; and determining, based on the comparing, that the captured image matches at least one of the candidate images.
 4. The method of claim 3, further comprising determining the occurrence of the condition by detecting that a global positioning system (GPS) trace of the client device has crossed within the threshold distance from the pickup location, wherein the candidate images include metadata indicating that the candidate images each correspond to a location that is within the threshold distance of the pickup location.
 5. The method of claim 2, further comprising: identifying, in the image, a visual indicator of the pickup location; generating persistent augmented reality content identifying the visual indicator; and placing the augmented reality content on a portion of the image where the visual indicator is located.
 6. The method of claim 2, further comprising: identifying, in the image, a visual indicator of a rider associated with the pickup location; generating persistent augmented reality content identifying the rider; and placing the augmented reality content on a portion of the image where the rider is located.
 7. The method of claim 1, further comprising: detecting, based in part on data obtained from a sensor of the client device, that a vehicle associated with the client device is stopped; and triggering display of the image in response to detecting that the vehicle is stopped.
 8. A non-transitory computer-readable storage medium storing computer program instructions executable by a processor to perform operations comprising: receiving confirmation from a client device of a driver that the driver will navigate to a pickup location; in response to receiving the confirmation, monitoring a location of the client device to determine whether the client device is within a threshold distance of the pickup location; determining, based on the monitoring, that the client device is located with the threshold distance of the pickup location; in response to determining that the client device is located within the threshold distance of the pickup location, selecting, based in part on a direction that the client device is facing, an image of one or more candidate images that each correspond to the pickup location; and instructing the client device to generate for display the image.
 9. The non-transitory computer-readable storage medium of claim 8, wherein the operations further comprise: generating persistent augmented reality content for display on the client device, the persistent augmented reality content associated with the pickup location; and instructing the client device to display the persistent augmented reality content as an overlay on top of the image.
 10. The non-transitory computer-readable storage medium of claim 8, wherein determining that the client device is within the threshold distance of the pickup location comprises: responsive to an occurrence of a condition, instructing the client device to capture an image as the client device approaches the pickup location; comparing a rendering of the captured image to the plurality of candidate images; and determining, based on the comparing, that the captured image matches at least one of the candidate images.
 11. The non-transitory computer-readable storage medium of claim 10, wherein the operations further comprise determining the occurrence of the condition by detecting that a global positioning system (GPS) trace of the client device has crossed within the threshold distance from the pickup location, wherein the candidate images include metadata indicating that the candidate images each correspond to a location that is within the threshold distance of the pickup location.
 12. The non-transitory computer-readable storage medium of claim 9, wherein the operations further comprise: identifying, in the image, a visual indicator of the pickup location; generating persistent augmented reality content identifying the visual indicator; and placing the augmented reality content on a portion of the image where the visual indicator is located.
 13. The non-transitory computer-readable storage medium of claim 9, wherein the operations further comprise: identifying, in the image, a visual indicator of a rider associated with the pickup location; generating persistent augmented reality content identifying the rider; and placing the augmented reality content on a portion of the image where the rider is located.
 14. The non-transitory computer-readable storage medium of claim 8, wherein the operations further comprise: detecting, based in part on data obtained from a sensor of the client device, that a vehicle associated with the client device is stopped; and triggering display of the image in response to detecting that the vehicle is stopped.
 15. A system comprising: a processor for executing computer program instructions; and a non-transitory computer-readable storage medium storing computer program instructions executable by the processor to perform operations for estimating a location of a client device, the operations comprising: receiving confirmation from a client device of a driver that the driver will navigate to a pickup location; in response to receiving the confirmation, monitoring a location of the client device to determine whether the client device is within a threshold distance of the pickup location; determining, based on the monitoring, that the client device is located with the threshold distance of the pickup location; in response to determining that the client device is located within the threshold distance of the pickup location, selecting, based in part on a direction that the client device is facing, an image of one or more candidate images that each correspond to the pickup location; and instructing the client device to generate for display the image.
 16. The system of claim 15, wherein the operations further comprise: generating persistent augmented reality content for display on the client device, the persistent augmented reality content associated with the pickup location; and instructing the client device to display the persistent augmented reality content as an overlay on top of the image.
 17. The system of claim 15, wherein determining that the client device is within the threshold distance of the pickup location comprises: responsive to the occurrence of a condition, instructing the client device to capture an image as the client device approaches the pickup location; comparing a rendering of the captured image to the plurality of candidate images; and determining, based on the comparing, that the captured image matches at least one of the candidate images.
 18. The system of claim 64, wherein the operations further comprise: identifying, in the image, a visual indicator of the pickup location; generating persistent augmented reality content identifying the visual indicator; and placing the augmented reality content on a portion of the image where the visual indicator is located.
 19. The system of claim 16, wherein the operations further comprise: identifying, in the image, a visual indicator of a rider associated with the pickup location; generating persistent augmented reality content identifying the rider; and placing the augmented reality content on a portion of the image where the rider is located.
 20. The system of claim 15, wherein the operations further comprise: detecting, based in part on data obtained from a sensor of the client device, that a vehicle associated with the client device is stopped; and triggering display of the image in response to detecting that the vehicle is stopped. 